RAG optimisation: answer quality (P1-P10), v17 numeric grounding, offline fallback, hybrid RPC hardening + golden-recall regression fix - #118
Merged
Conversation
…ieval hardening Retrieval / database: - De-drift and fix all four hybrid RPCs (ambiguous-id 42702 + plpgsql seqscan 130s regression); migrations for chunks/index-units/embedding-fields/memory-cards, schema-health execution smoke, dead-variant cleanup, and a pg_trgm clinical-term typo-correction function - Structured hybrid-RPC failure logging + telemetry (hybrid_rpc_errors) so silent layer death can never hide again; golden retrieval eval expanded 10 -> 23 cases (23/23 green) - OR-relaxation + trigram typo-correction fallbacks for over-conjunctive tsqueries (8a/8b) Answer generation quality (P1-P10): - Synthesis prompt rewritten: targeting-first, positive clinician voice, few-shot examples, strict numeric verbatim rules (no invented ranges/levels); cache key v13 -> v17 - Responsiveness gate extended to model answers on simple direct non-definition questions - Natural generation-timeout fallback wording; offline extractive phrasing softened - Query-class-aware strong reasoning effort (high only for dose/threshold) + 30s answer timeout; procedural "what is required" queries no longer false-fail the definition fragment gate - Number-safe, sentence-boundary source truncation for model context (truncateForModel) - Unverified figures are un-bolded and the faithfulness caveat is deduplicated (single CRITICAL gap); per-document crowding cap (3 chunks/doc) on model context including retry paths - Values-only bolding policy; flattened-table run-on separation safety net Offline / provider resilience: - RAG_PROVIDER_MODE=auto|openai|offline with embedding-free lexical retrieval, fail-closed source-only answers, answerQualityTier labels, and UI source-only disclosure - insufficient_quota distinguished from transient rate limits; auto-degrade on provider failure Validated: verify:cheap 706 passed; retrieval golden 23/23 (recall 1.0); eval:rag 15/15 grounded; eval:quality grounded-supported 0.833 -> 0.933, numeric warnings 0.175 -> 0.125 (v17 probe: 6 -> 1 unverified tokens on regression queries); p95 latency 54s -> ~17s; offline proof 0 OpenAI tokens. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…factor with worktree P1-P10 Resolutions: - applyNumericVerification: combined policies — keep this branch's un-bolding of unverified figures and single-caveat dedupe, adopt main's hard fail-closed gate for unverified numbers in actionable dose/threshold/monitoring context (numeric_faithfulness_gate_source_gap) - generateWithModel: keep options-object shape, query-class-aware strong reasoning effort, and P9-capped retry contexts; adopt main's broader truncation retry (strong-route retries too); drop main's now-dead modelTierUsed in favour of usedStrongModel/retriedWithStrong flags - sourceBackedGenerationTimeoutAnswer: keep this branch's natural source-pointer wording (validated against the source-inventory quality detector) - env/.env.example: keep documented non-pro strong tier + 30s answer timeout; adopt main's runtime pro-model guard (runtimeAnswerModel) via requestedOpenAIAnswerModels import - tests: rag-answer-fallback withhold assertion made bold-marker tolerant; content-accuracy tests updated for the merged two-tier numeric policy (caveat path + hard-gate path) verify:cheap green after merge: 736 passed / 2 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ecall 0.76 -> 1.0 The merged main change to retrieval-selection.ts (uncapped candidate score + blanket source- governance metadata weighting) regressed the 23-case golden retrieval eval to 16/23 (doc-recall@5 1.0 -> 0.76, mrr 0.75 -> 0.64). Root cause, measured by bisection: - live hybrid/finalScore values saturate at 1.0, so uncapping let boost stacking fully override lexical relevance ordering within the saturated region - the corpus is only partially metadata-enriched; normalizeSourceMetadata coerces unenriched documents to unknown/unverified, so blanket status penalties (-0.21) vs enriched boosts (+0.14) swung ranking by ~0.35 for reasons unrelated to relevance and buried correct documents Fix: restore the clamp on the candidate relevance score and remove governance metadata weighting from selection ordering entirely (intent boosts retained). Governance stays enforced by ranking penalties and the answer/source-governance layer; RC8 (source-strength filtering) remains tracked in docs/rag-hybrid-findings-and-todo.md. Amended the retrieval-selection governance test to the corrected contract with the measured justification. Validated: golden retrieval 23/23 (doc-recall@5 1.0, content-recall@5 1.0, top-k 1.0, mrr 0.743); retrieval-selection tests 7/7; verify:cheap 736 passed / 2 skipped; eval:rag 5/5 grounded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dentally resurrected DashboardFloatingFab.tsx, ui/button.tsx, ui/icon-button.tsx, ui/skeleton.tsx, and lib/tools.ts were deleted on main (component extraction/cleanup); a git checkout during retrieval-regression bisection restored the old copies into the worktree and they were committed unintentionally. No current source imports them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 2, 2026
Merged
BigSimmo
added a commit
that referenced
this pull request
Jul 2, 2026
…ata debt at 0.6 (#176) * test(eval): scope danger-warning failures to answered routes; re-accept metadata debt Two release-quality-eval governance fixes, no app-behavior change. 1. Danger source-governance warnings on unsupported (declined) routes are no longer counted as failures. The app attaches a weak-evidence danger warning when relevance verdict is "none", which on a refusal is the correct reader signal ("no source-backed evidence, do not act"), not a governance failure. The metric now counts danger warnings only on answered routes (a delivered answer standing on dangerous sourcing). For genuinely out-of-scope cases the warning's presence is positively asserted so a silent regression is caught; for supported questions that were wrongly refused, the routing failure is reported elsewhere and no misleading missing-warning noise is added. Verified on the live release eval: source_governance_danger_failure_rate 0.225 -> 0, all 9 affected cases are correctly-declined out-of-scope queries. 2. Re-accept the bounded source-metadata debt at a 0.6 review-required ceiling (expiry unchanged, 2026-07-31). The rate rose 0.14 -> 0.5398 with no corpus change: the relevance-first ranking work (PR #118 / PR #130) removed governance metadata weighting from selection ordering, so review-flagged sources are no longer buried and the metric now reports true corpus state. docs/source-review-priority-2026-07-02.md lists the 25 documents occupying golden top-5 slots, highest-frequency first; reviewing the top 12 returns the rate to ~0.12, after which the ceiling should be tightened back toward 0.2. Note: the eval still reports separate blocking failures (grounded_supported, one prompt-injection false-positive, citation, extractive latency) that are pre-existing answer-quality regressions unrelated to these two changes and tracked separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(eval): key danger-warning failures on grounded, not route (review fixes) Addresses two automated review notes on the danger-warning reclassification: - Exempt non-grounded refusals however they arise. finalizeRagAnswerQualityCore can convert a fast/strong/extractive answer into an evidence-gap refusal by setting grounded=false while preserving the original routingMode, so a route-based check could still fail the gate on a declined answer. Key the danger-warning failure on grounded === true (an answer was actually delivered) instead of route !== "unsupported". - Drop the toothless positive "missing danger warning" assertion. It only surfaced in failed_cases and never fed a thresholded metric, so it could not gate; making it gate would introduce new blocking failures on the current corpus (e.g. close-title-noise), which we explicitly must not do. The grounded-based check is the clean, robust core. Behavior unchanged on the current corpus: danger rate stays 0 (all danger warnings are on grounded=false refusals). Unit tests updated to cover the finalize-core evidence-gap refusal path. vitest 8/8, typecheck + lint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 6, 2026
14 tasks
BigSimmo
added a commit
that referenced
this pull request
Jul 14, 2026
Audit 2026-07-13 deferred items D4 + D5, completed as zero-behavior-change plumbing per the #118 lesson (no governance weighting without golden-eval proof) — the levers ship OFF and are activated by the operator only behind a green golden eval once OpenAI quota recovers. - D4: secondStage.unknownCurrentnessPenalty in ranking-config (default 0, clamped non-negative) applied in the second-stage rerank when document_status is 'unknown'. Activation = RAG_RANKING_CONFIG JSON override, the module's designed eval-gated tuning path. - D5: deriveTrust broadening behind NEXT_PUBLIC_RAG_TRUST_CAP_ALL_CLAIMS (unset = false): when on, unverified-authority evidence caps trust to medium for ALL supported claims, not just high-risk ones. NEXT_PUBLIC because answer-render-policy renders client-side. - Tests lock both the zero-change defaults and the flag-on behavior. Verified: verify:cheap green (2,310 tests), eval:rag:offline green. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 20, 2026
Merged
BigSimmo
pushed a commit
that referenced
this pull request
Jul 30, 2026
BigSimmo
pushed a commit
that referenced
this pull request
Jul 30, 2026
…able Every open PR sampled carries the same ci/circleci: verify failure, including #1400 which is a docs-only AGENTS.md change, so it is not branch-specific. The job's whole contents were mirrored locally on #1396's tip and all of it is green, including the PyMuPDF-gated PDF tests under a venv built exactly as .circleci/config.yml builds it. That puts the fault in the job environment. Recorded with the quota hypothesis marked explicitly unverified: the CircleCI project is private, no session has a CircleCI token, and the unauthenticated API returns "Build not found". Reading the failing step needs an operator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01361jh3eYVjJCzXWjAhdZiF
BigSimmo
added a commit
that referenced
this pull request
Jul 30, 2026
… PR babysitting (#1421) * issues: capture two CI/merge operational findings from PR babysitting #117: this repo's CI (on: pull_request with default types) doesn't retrigger on the draft-to-ready transition, only on opened/synchronize/reopened — a marked-ready PR can sit with a minimal check set until an actual new commit lands. #118: GitHub's update-branch API doesn't honor the merge=union .gitattributes driver on docs/branch-review-ledger.md, so it can 422 with a false conflict that a local git merge resolves cleanly. Both observed today on PR #1406. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Re9ERtQwJ82ErbAnahAhsa * Address Codex review: clarify ledger rows #125/#126 wording #125: spell out that ready_for_review must be added alongside the existing pull_request.types defaults, not replace them. #126: qualify the update-branch fallback push with the same explicit user confirmation the provider boundary already requires. * docs(issues): capture #1396's unaddressed physical-device chrome gate PR #1396 repeatedly declared physical-device Safari/PWA acceptance (docs/phone-chrome-physical-acceptance.md) as required before merge because headless Chromium cannot certify Safari chrome minimisation or cold-launch PWA paint, then merged with the checklist still blank. Also notes a related missing pre-paint/cold-load hydration test the same PR's review flagged but never filed. * style: fix table padding drift from the main merge npx prettier --write after merging main (9e2fe44) — a table cell width shifted during the merge and format:check would have caught it. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
BigSimmo
pushed a commit
that referenced
this pull request
Jul 30, 2026
Fourth genuine conflict on docs/outstanding-issues.md, confirmed with git merge-tree before acting. Main's #1470 rewrote #13 as measurement-gated while this branch changed #17, #105 and added #147. Verified by hash that main touched neither #17 nor #105 — both are byte-identical at the merge base and at origin/main — so those rows take this branch's version and #13 takes main's, rather than either side winning wholesale. Also de-duplicated against main's new #118, which already owns adopting the Lighthouse baseline and flipping enforce. #147's gate paragraph now cross-references it instead of restating it, and contributes the constraint #118 was missing: pinning a baseline today would bake CLS 0.363 on /dsm in as the accepted value, so the CLS fixes must land first or the gate ratifies the breach. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01361jh3eYVjJCzXWjAhdZiF
BigSimmo
pushed a commit
that referenced
this pull request
Jul 31, 2026
Resolves the fifth conflict on docs/outstanding-issues.md by taking main's side on both hunks, after verifying that was correct rather than convenient: main renumbered the queue to fix a duplicate rank 18 this branch introduced, and it archived #138 and #146 rather than dropping them (confirmed present in the archive table). The resulting id set is byte-identical to main's. #150 escalated. That row records CodeRabbit reviewing none of a full day's PRs and rests explicitly on the Codex connector being the surviving reviewer that "found three real defects that had survived local gates". On PR #1505 Codex posted its own usage-limit notice alongside CodeRabbit's spending-cap notice, so that PR received zero automated review and so will anything opened while both caps hold. This is the same issue with its fallback removed, not a new one — which matters because the single Codex finding on #1459 was correct and caught a verification that had matched the wrong component, closing #105 on bad evidence. Local gates did not catch it. The attribution harness lands as scripts/measure-cls-attribution.mjs. #147's next step needs a before/after CLS pair, and #118 will want element attribution again; without this, both mean re-deriving it. Build and serve mirror run-lighthouse-budget.mjs so the numbers sit beside the Lighthouse reports. Two traps that cost real time are encoded rather than left to be rediscovered: CHROME_PATH must be set where the browser is outside a standard location, and an init script attaching a MutationObserver to document.documentElement before <html> exists throws and silently takes the CLS observer with it — so the script now fails loudly on an all-zero result instead of reporting a false clean bill. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01361jh3eYVjJCzXWjAhdZiF
BigSimmo
added a commit
that referenced
this pull request
Aug 4, 2026
These came out of the DS V2 wave with no ledger row and would otherwise have evaporated with the session: the stale PR-J governance review record, the COMPONENTS.md section 0 drift plus the undocumented optionality-marker contract change, the undocumented answer-copy-payload module, the missing ADOPTION.md section 7 proof shots, and the two orphan builder branches and leftover dev servers. The sixth residual, visual baselines staying zero and advisory, is already ledger #118 and was deliberately not duplicated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cursor Bot
pushed a commit
that referenced
this pull request
Aug 4, 2026
…o states that need one (#1612) * docs(issues): record #230 live answer generation timeout Production triage 2026-08-04 measured the live failure: OpenAI generation overruns OPENAI_ANSWER_TIMEOUT_MS and the answer degrades to source-only, while retrieval passes its gate with 12 candidates across 5 documents. The 2026-08-04 handover's four ranked hypotheses (query embedding, Supabase RPC, source governance, corpus) are all ruled out by that evidence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(ds): let EmptyState carry a heading, and adopt the two states that need one EmptyState rendered its title as a <p>, so any empty state that owns a region of the page could not adopt it without losing its heading. /dsm/search lost exactly that when it adopted the primitive (#224), and the main document-search empty state stayed hand-rolled because ui-smoke pins a heading role on it (#217). Add an opt-in headingLevel prop. It is deliberately un-defaulted: most adopted call sites sit inside a card that already owns its region's heading, and promoting every title would inject an outline level the page never declared. - /dsm/search passes headingLevel={2}, restoring the heading it had before. - The document-search empty state adopts EmptyState with headingLevel={3}, which also converts it from a bespoke centred card to the shared state surface and gives it the polite live region every other adopted empty state has. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(issues): correct #230 — the fast route budget binds, not the env timeout The first row named OPENAI_ANSWER_TIMEOUT_MS as the constraint. It is not, and production does not set it. The query routes fast, so answerRouteBudgetMs.fast (25s) is the ceiling and generationRequestTimeoutMs holds back the 2s recovery reserve on top; the measured 24.7s is that budget being spent. Also records the half with clinical consequence: answerRouteResultCanBeCached excludes a route-deadline-exceeded answer but not one degraded by the OpenAI request timing out, so a generation_fallback answer can be cached and re-served after the provider recovers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(ledger): record the PR #1612 review Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(issues): capture the five Wave 5 close-out residuals (#231-#235) These came out of the DS V2 wave with no ledger row and would otherwise have evaporated with the session: the stale PR-J governance review record, the COMPONENTS.md section 0 drift plus the undocumented optionality-marker contract change, the undocumented answer-copy-payload module, the missing ADOPTION.md section 7 proof shots, and the two orphan builder branches and leftover dev servers. The sixth residual, visual baselines staying zero and advisory, is already ledger #118 and was deliberately not duplicated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(issues): archive #217 and #224 completed by EmptyState headingLevel PR #1612 added the opt-in headingLevel prop and adopted the DSM and document-search empty states that needed it. Close both ledger rows so /issues stops treating finished work as outstanding. * docs(issues): correct #231 summary to name the binding fast-route budget The detail already established that OPENAI_ANSWER_TIMEOUT_MS is not the binding ceiling — answerRouteBudgetMs.fast = 25000 is. Align the summary so triage does not raise a non-binding setting, and backtick the src/lib/rag/** path so markdownlint MD037 stays quiet. * ci: retrigger Production UI after known Services viewport-shrink flake Production UI (1) failed once on ui-phone-scroll-page-owned:577 (Services results re-settle after viewport shrink) — same signature as the 2026-07-31 flake noted in the branch-review ledger. Unrelated to EmptyState headingLevel; shards 2/3 and critical passed. Cannot rerun-failed via this token (403), so retrigger the pull_request run. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
BigSimmo
added a commit
that referenced
this pull request
Aug 8, 2026
Same defect Codex flagged on PR #1719: the commits recording these measurements were authored 2026-08-08 (07:xx UTC), so a 2026-08-09 stamp places every re-measurement after the commit that recorded it. Four occurrences, in the #118, #269 and #270 rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
7 tasks
BigSimmo
added a commit
that referenced
this pull request
Aug 8, 2026
cursor Bot
pushed a commit
that referenced
this pull request
Aug 8, 2026
Codex P2 on PR #1724: the red advisory is a stale Chromium 150 baseline overwrite from #1710 against a job already pinned to Playwright Chromium 151 (#1716), not a fresh CI-runner 150→151 bump. Prefer restoring or comparing the known 151 baseline over a blind --update. Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
BigSimmo
added a commit
that referenced
this pull request
Aug 8, 2026
The first commit adopted the six PNGs but missed `tests/design-system-adoption.test.ts`, a committed contract that governs exactly this adoption and that no handover in this series mentions. It was dormant only because it had no committed baselines to check. `npm run verify:cheap` caught it: 9 failed / 5586 passed. Three things it requires, all now met: - `tests/__screenshots__/linux/provenance.json` (schemaVersion 2) binding the goldens to their capture: platform `linux`, runner `ubuntu-24.04`, hosted-CI artifact `visual-baseline-31251091603`, candidateSourceHead `bc33d414e`, and a SHA-256 plus pixel dimensions per candidate. The reviewer attestation is a human one — recorded after the six PNGs were actually looked at, not asserted on their behalf. - `AWAITING_BASELINE` must stay a *static literal* `new Set([...])`, and the suite may differ from the capture commit in nothing but those values. The explanatory comment added in the first commit violated that, and `new Set<string>()` did not parse as a literal set at all. Both reverted; the suite diff is now exactly the six ids. - Only an allowlisted set of paths may change alongside the adoption, which is why the `tests/__screenshots__/README.md` note from the first commit is reverted here. The document-viewer limitation it recorded is in the PR description instead. The fixture change is the one substantive fix. `initialiseCandidateRepository` seeded its candidate-source commit from the *live* `tests/ui-visual-baseline.spec.ts`, so every fixture silently depended on the repository not having adopted its baselines yet — and the commit that empties `AWAITING_BASELINE`, the outcome the contract exists to permit, made the fixture's own candidate head fail "must contain exactly the canonical six ids". It now defaults to the explicit canonical list. A fixture states its precondition rather than borrowing it from the tree under test. Verified: tests/design-system-adoption.test.ts 51 passed (51). Mutation-tested by corrupting a provenance SHA-256, which fails with the expected mismatch. Refs #118
cursor Bot
pushed a commit
that referenced
this pull request
Aug 8, 2026
The superseded banner said all six visual baselines were adopted under #118, but the ledger still has #118 open and tests/__screenshots__/ holds only README.md. Keep the accurate Ubuntu-CI generation point without implying visual-regression protection exists yet. Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
BigSimmo
added a commit
that referenced
this pull request
Aug 8, 2026
* docs(design-system): stamp the 7 August handover superseded Nine open ledger rows (#261, #262, #264-#270) cite docs/design-system/HANDOVER-2026-08-07.md as their Source, and docs/design-system/README.md sent anyone picking the work up cold straight to it as "measured state, the ordered plan". Four of its figures have since been disproved, and the corrections were written into the ledger rows rather than the document, so the document still asserts the originals. A session scoping from it re-derives work that is already known wrong. Recorded as #277. A banner rather than a rewrite, which is what #277 asks for: the corrections already live in the rows, and duplicating them re-creates the drift this fixes. The banner names the four measured errors -- the "229 --shadow-tight aliases" that is really a seven-token total mislabelled as one token (100 sites across 55 files, 228 total), the 24-vs-23 unadopted count, the "baselines cannot be generated on Windows" conclusion that overlooked the ubuntu CI job now used under #118, and #270's 22-site premise that has zero same-variant pairs -- then points at the rows. The README now sends a cold start to docs/outstanding-issues.md and marks the handover superseded in the link text itself, where the misdirection was. Deliberately not done: the document is not deleted, moved, or corrected in place. The nine Source citations, the PR and commit record, and its verification and gotcha sections are provenance the ledger is meant to preserve, and silently correcting it would leave those rows citing a document that no longer says what they were derived from. Verified: npm run docs:check-links -- "docs link check passed: 1667 repo path references resolve"; npm run docs:check-inventory -- current; npm run format:check -- "All matched files use Prettier code style!". Refs #277 * docs(design-system): drop false #118 baseline-adoption claim The superseded banner said all six visual baselines were adopted under #118, but the ledger still has #118 open and tests/__screenshots__/ holds only README.md. Keep the accurate Ubuntu-CI generation point without implying visual-regression protection exists yet. Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
BigSimmo
added a commit
that referenced
this pull request
Aug 8, 2026
…provenance contract (#1729) * test(visual): adopt the six linux visual baselines from CI run 31251091603 `tests/__screenshots__/` has held only a README since the visual gate landed, so every target sat in `AWAITING_BASELINE`, skipped rather than compared, and no PR could prove it had not moved the look. That is the blocker `#118` records and the one `#269` is formally waiting on. Adopted from the ubuntu CI artifact `visual-baseline-31251091603` — a `push` on `main` at `bc33d414e`, never from a developer machine: `snapshotPathTemplate` is platform-scoped, so win32 PNGs land in `win32/` and the ubuntu job never reads them, and font hinting alone would make every later run red. All six goldens were reviewed before committing, and stability was measured rather than assumed: five of the six are byte-identical by SHA-256 to the candidates from the earlier independent run 31249978408 at `2069b1f5c`. Only `document-viewer` differs, and those two runs straddle #1705, which changed `DocumentViewer.tsx`. `AWAITING_BASELINE` is now empty, so a missing golden is once again unambiguously a fault rather than a declared exemption. The two honesty tests keep that list from rotting in either direction. Deliberately NOT included: promotion. `visual-baseline` stays `continue-on-error` and outside `pr-required`. The workflow comment asks for a soak first, and adding the job to `pr-required` and dropping `continue-on-error` belong in one later edit. Known limitation recorded in the README rather than papered over: the `document-viewer` target clips a ~2900px `#main-content` against a 900px viewport and contains viewport-pinned chrome (`sm:sticky` header, `sm:fixed` composer), which composites partway down the stitched capture and overlaps content. It is deterministic, so the comparison still means something, but it predates #1705 and would be quieter with a narrower clip or a mask. Refs #118 * test(visual): satisfy the committed baseline provenance contract The first commit adopted the six PNGs but missed `tests/design-system-adoption.test.ts`, a committed contract that governs exactly this adoption and that no handover in this series mentions. It was dormant only because it had no committed baselines to check. `npm run verify:cheap` caught it: 9 failed / 5586 passed. Three things it requires, all now met: - `tests/__screenshots__/linux/provenance.json` (schemaVersion 2) binding the goldens to their capture: platform `linux`, runner `ubuntu-24.04`, hosted-CI artifact `visual-baseline-31251091603`, candidateSourceHead `bc33d414e`, and a SHA-256 plus pixel dimensions per candidate. The reviewer attestation is a human one — recorded after the six PNGs were actually looked at, not asserted on their behalf. - `AWAITING_BASELINE` must stay a *static literal* `new Set([...])`, and the suite may differ from the capture commit in nothing but those values. The explanatory comment added in the first commit violated that, and `new Set<string>()` did not parse as a literal set at all. Both reverted; the suite diff is now exactly the six ids. - Only an allowlisted set of paths may change alongside the adoption, which is why the `tests/__screenshots__/README.md` note from the first commit is reverted here. The document-viewer limitation it recorded is in the PR description instead. The fixture change is the one substantive fix. `initialiseCandidateRepository` seeded its candidate-source commit from the *live* `tests/ui-visual-baseline.spec.ts`, so every fixture silently depended on the repository not having adopted its baselines yet — and the commit that empties `AWAITING_BASELINE`, the outcome the contract exists to permit, made the fixture's own candidate head fail "must contain exactly the canonical six ids". It now defaults to the explicit canonical list. A fixture states its precondition rather than borrowing it from the tree under test. Verified: tests/design-system-adoption.test.ts 51 passed (51). Mutation-tested by corrupting a provenance SHA-256, which fails with the expected mismatch. Refs #118
BigSimmo
added a commit
that referenced
this pull request
Aug 8, 2026
… not only the first adoption (#1742) `validateCandidateSourceBinding` required `candidateSourceHead` to name a commit whose `AWAITING_BASELINE` still declared the canonical six, and the current tree to declare none. That models the FIRST adoption and nothing else. After #1729 landed, no commit on main declares the six again, so no future capture head could ever satisfy it -- the binding was satisfiable exactly once, and that once has passed. The consequence is only latent today, and this commit does not pretend otherwise: `validateLinuxVisualBaselineSet` short-circuits on `declaredPaths.length === 0` and no surface yet declares `baseline.status: "committed"`, so the whole contract is dormant and refreshing a golden is currently ungated. The defect bites the moment those surfaces are declared (#242) -- at which point the first intentional design change would leave the goldens red with no supported way to re-adopt them, which is precisely backwards for a gate meant to make design changes provable. So the binding now accepts either shape: FIRST ADOPTION candidate head declares the six, current declares none. REFRESH both ends empty, suite byte-identical. A refresh is the ordinary case once baselines exist: a surface is deliberately re-shot and its goldens replaced while the suite itself does not change at all. Normalisation already blanks the AWAITING values on both sides, so the "suite changed beyond the AWAITING declaration" check stays exact for both. Nothing is loosened beyond that. A candidate head with five ids, seven ids, a dynamic set, a spread or a duplicate still fails, as do a non-empty current list, a non-ancestor head, a hash mismatch, a wrong platform and an unreviewed set. The one previously-untested shape -- a refresh -- now has its own case, because the gap was invisible precisely because nothing exercised it. Verified: tests/design-system-adoption.test.ts 51 passed (51), including the new refresh case; check:design-system-adoption and check:design-system-contract exit 0; format:check clean. Refs #242, #118
cursor Bot
pushed a commit
that referenced
this pull request
Aug 8, 2026
…seline, and add a baseline adopt helper (#1743) * feat(design-system): mask the pinned chrome in the document-viewer baseline, and add a baseline adopt helper Two changes that make refreshing a golden after a deliberate design change cheap and readable, which is what the pixel gate needs in order to be read at all. Mask the viewport-pinned chrome (#278). The document-viewer target clips a ~2900px region against a 900px viewport, and it contains a `sm:sticky sm:top-0` header and a `sm:fixed` composer. Playwright stitches an oversized element capture, so both land partway DOWN the image, overlap whatever content sits behind them at that offset, and move whenever content above them changes height -- so an unrelated edit anywhere on the page redrew two bands of the golden and inflated every diff. Masked rather than clipped away, because both are real chrome that belongs in the frame and narrowing the selector would drop the rail panels this target exists to watch. Their own geometry is covered by the phone-chrome contracts in docs/search-chrome-behaviour.md, not by this pixel gate. Fail loudly when a mask matches nothing. A mask selector that matches no element masks nothing, silently -- the golden keeps comparing the region the mask was meant to exclude while the declaration reads as protection that is not there. Renaming a class is enough to cause it. Every declared mask must now resolve to at least one element before the comparison is trusted. That guard is what makes the mask above safe to rely on rather than merely present. Add scripts/adopt-visual-baselines.mjs (npm run design-system:baselines:adopt). Adopting previously meant hand-copying six PNGs and hand-assembling provenance.json with a SHA-256 and pixel dimensions per candidate, the capture commit, the run id and the reviewer attestation. Doing that by hand on every design change is the friction that makes people skip the refresh and leave a red advisory standing. It resolves candidates from both shapes the artifact can take -- visual-candidates/ when a target was awaiting a baseline, and <id>-actual.png when it compared and differed. It refuses a missing or non-numeric run id, a short or unknown capture commit, and a missing --reviewed-by, because that field records a HUMAN review of the images. It never captures screenshots: baselines are platform-scoped and a developer-machine shot lands where ubuntu CI never reads it. Dry run by default. The document-viewer golden is NOT refreshed here -- masking changes its pixels, so it must be re-shot from a CI run that already contains this mask. That is the refresh loop the helper exists for, and it needs this to land first. Verified: tsc 0 errors; lint exit 0; check:design-system-adoption exit 0; docs:check-inventory current at 232 npm scripts; docs:check-links exit 0; format:check clean. Helper exercised against two real artifacts -- it resolves all six candidates from run 31251091603 and correctly refuses run 31254917796, which was all-green and so contains no candidate images -- and all three input guards were confirmed to reject. Dry run wrote nothing. Refs #278, #118 * fix(design-system): support partial visual baseline refresh adoption When CI changes only some surfaces, passing targets emit no *-actual.png and unchanged images live under tests/__screenshots__/ in the artifact. The adopt helper now resolves candidates from diff output, artifact baselines, or committed goldens, records refresh provenance (including replacedCandidateIds), and validates capture-head AWAITING_BASELINE binding before writing. Also fix git cat-file reachability checks and run mask-selector validation before candidate capture. * chore(design-system): refresh document-viewer baseline after chrome masks Adopt the masked document-viewer capture from visual-baseline-31267928439 (tip ac1d0dd) via the partial-refresh helper. Clears the expected advisory diff introduced when the sticky header and fixed composer were masked. Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com> * fix(design-system): gate retained baselines on junit and uniquify the mask Copilot: retaining screenshots from the artifact without visual-junit proof could bless a stale golden for a target that failed before emitting an actual. Require a passing junit case for every retained id, refuse all-green refreshes, and only treat *-actual.png as a fresh diff. Also mask [data-document-sticky-header] instead of .edge-glass-header so the fail-loud guard cannot pass against the universal search header after a DocumentViewer rename. Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com> * chore(design-system): refresh masked document-viewer baseline after main sync Re-adopt from visual-baseline-31268982766 on the post-#1741 merge tip so the golden includes the unique data-document-sticky-header mask and the viewer rework from main. Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
cursor Bot
pushed a commit
that referenced
this pull request
Aug 8, 2026
) * ci(visual): stop running the visual baseline job on pull requests Owner decision, 2026-08-09: the pixel gate taxes exactly the work it exists to support. During a redesign the honest result is red on every UI pull request, and the baselines cannot be refreshed until a run has produced the new images, so that red is unavoidable rather than actionable. It was never merge-blocking -- `continue-on-error: true`, and absent from `pr-required`, whose needs list is [changes, static-pr, safety, coverage, build, container-images, ui-critical-fast, ui-critical, db-reset-verify]. But a red check nobody can act on mid-iteration is friction whether or not it gates, and it generated a steady stream of failure notifications on PR #1743 while the only correct response was "yes, that is the change". Drift is still caught. The job keeps running on pushes to main and release branches, on the weekly schedule, and on demand via workflow_dispatch. That moves the signal to where a refresh is cheap -- after the change has landed, against a run whose artifact already contains the new images -- instead of interrupting the change. One command refreshes it: npm run design-system:baselines:adopt -- --from <artifact> --run-id <id> --head <sha> --reviewed-by "<name>" --write Nothing else changes: the job, its targets, its provenance contract and its advisory status are untouched, and no required check is added or removed. Verified: npm run check:github-actions passed; ci-cache-safety 18 passed, 11 skipped; no test pins this job to pull_request; format clean. Refs #118 * issues: record the owner decision that visual-baseline stays off pull requests * fix(ci): keep visual-baseline off merge_group as well as PRs Allowlist push/schedule/workflow_dispatch so merge-queue UI churn does not re-run the advisory pixel gate, and document the schedule in testing.md. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: BigSimmo <BigSimmo@users.noreply.github.com>
BigSimmo
pushed a commit
that referenced
this pull request
Aug 12, 2026
…re-scope the wave plan Archived with source evidence: - #166 the verify-before-use caveat ships (verification-notice.tsx:52, rendered unconditionally by AnswerCard at answer-card.tsx:108). I reported this row as open twice before reading the code; the archive says so. - #208 buildAnswerClipboardText composes renderCopyText rather than replacing formatAnswerRenderCopyText (answer-copy-payload.ts:121-138). - #216 AnswerCard is adopted on the answer surface and the manifest agrees. Re-scoped #250: its wave plan names A1 rows that are now closed and hygiene rows that are in flight, so as written it misdirects. Recorded what actually remains (#147, #117, #118, #98, #189). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017paT42ZVMf8jaLtkjFxdy5
BigSimmo
pushed a commit
that referenced
this pull request
Aug 12, 2026
Measured, not assumed. Two independent runs of the offline harness this row specifies, on Chromium 141 — the same browser that produced the row's original figures, so before/after is like-for-like. Mobile CLS (run1/run2 vs recorded): /dsm 0.035 / 0.035 was 0.363 /documents/search 0.000 / 0.000 was 0.220 / 0.013 / 0.013 was 0.023 /forms 0.081 / 0.081 was 0.310 local /therapy-compass 0.000 / 0.000 was 0.240 local The 128px reserve round trip is gone. The cause was fixed by PR #1616, not by this session: use-phone-overlay-chrome-reserve.ts keeps the CSS seed until an observer candidate holds across an 80ms quiet window and publishes only via ResizeObserver — both remedies the row proposed — and its comment at :61 cites the row by number. Two controls make this a pass rather than the false clean the row warns about: values are not uniformly 0.000, so the observer is live; and the /dsm variance the row recorded (0.363 vs 0.219) has itself vanished, which is what you expect once the transient write causing it stopped. Also recorded: #118 updated — the Lighthouse browser drift is 141-vs-151, wider than the 150-vs-151 the row states, so the budget cannot grade from this container at all. New row #308 for desktop /documents/search CLS 0.119, stable against a 0.119 baseline, above threshold, and outside #147's mobile scope — which also makes the row's "desktop passes everywhere 0.016-0.097" stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017paT42ZVMf8jaLtkjFxdy5
BigSimmo
added a commit
that referenced
this pull request
Aug 17, 2026
* docs(issues): queue inbox request for mobile CLS regression on / PR #2050's Lighthouse budget check surfaced a real, reproducible mobile-root CLS regression (0.236 vs the committed 0.013 baseline) unrelated to that PR's diff. Cross-references the previously-resolved #147/#118 CLS work so it isn't lost; reconciliation into the canonical ledger happens separately. * docs(ledger): record review for PR #2059 (CLS regression issue log) --------- Co-authored-by: Claude <noreply@anthropic.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RAG_PROVIDER_MODE=auto|openai|offline, embedding-free lexical retrieval, fail-closed on weak evidence,answerQualityTierlabels + UI disclosure;insufficient_quotadistinguished from transient rate limits.retrieval-selection.tsregressed the golden eval to 16/23 (doc-recall@5 1.0→0.76) on the partially-enriched corpus. Restored relevance-first clamped ordering; governance stays enforced in ranking penalties and the answer layer. Governance selection test amended with measured justification.origin/maintwice (incl. PR Codify live retrieval RPCs, reclaim dead vector indexes, unify schema health #117's codified live RPCs — confirmed those snapshots carry the fixedlanguage sqldefinitions).Measured results: golden retrieval 23/23 (doc/content-recall@5 1.0, mrr 0.743) · eval:rag grounded 15/15 · eval:quality grounded-supported 0.833→0.933 · numeric-warning rate 0.175→0.125, v17 probe 6→1 unverified tokens · p95 latency
54s→17s · offline proof: 0 OpenAI tokens, grounded source-only answers, fail-closed on weak evidence.Verification
npm run verify:cheap(736 passed / 2 skipped, lint + typecheck clean)npm run verify:ui(UI change is a single disclosure note; covered by unit tests, Chromium run deferred to protect API budget)npm run verify:releasenpm run format:checknpm run check:production-readiness(READY; Supabase target confirmed)Clinical Governance Preflight
Clinical KB Database(sjrfecxgysukkwxsowpy)Notes
docs/rag-hybrid-findings-and-todo.md(RC8 source-strength filtering, ef_search, targeting-metric eval wiring).🤖 Generated with Claude Code